Learning of Uncontrolled Restless Bandits with Logarithmic Strong Regret

In this paper we consider the problem of learning the optimal dynamic policy for uncontrolled restless bandit problems. In an uncontrolled restless bandit problem, there is a finite set of arms, each of which when played yields a non-negative reward. There is a player who sequentially selects one of the arms at each time step. The goal of the player is to maximize its undiscounted reward over a time horizon T . The reward process of each arm is a finite state Markov chain, whose transition probabilities are unknown to the player. State transitions of each arm is independent of the player’s actions, thus “uncontrolled”. We propose a learning algorithm with near-logarithmic regret uniformly over time with respect to the optimal (dynamic) finite horizon policy, referred to as strong regret, to contrast with commonly studied notion of weak regret which is with respect to the optimal (static) single-action policy. We also show that when an upper bound on a function of the system parameters is known, our learning algorithm achieves logarithmic regret. Our results extend the literature on optimal adaptive learning of Markov Decision Processes (MDPs) to Partially Observed Markov Decision Processes (POMDPs). Finally, we provide numerical results on a variation of our proposed learning algorithm and compare its performance and running time with other bandit algorithms.